Learned heuristics: imitate solved plans, then optimise search cost directly - #157
Conversation
Every solved instance is already a labelled trajectory: the cost of a
plan's suffix from a state on it is that state's cost-to-go. `jupyddl
learn` turns a corpus of those into a heuristic and then stops imitating
h* and starts optimising the thing that actually matters.
Trained on 3-6 block instances, evaluated on 9-13 block instances from a
seed family no stage of training saw, under greedy best-first search:
heuristic coverage expanded seconds cost
learned 1.00 137 0.038 48.2
hff 1.00 518 0.561 51.8
goalcount 1.00 2483 0.169 51.0
Three design decisions did most of that work.
**Features are keyed on the predicate symbol, never the ground atom**,
and normalised per symbol. That is the entire transfer story: a one-hot
over ground atoms changes length and meaning with every instance, so a
model trained on four blocks cannot even be evaluated on forty.
**The objective is ranking, not regression.** Greedy best-first search
never reads a heuristic value, only the order it imposes; a model
uniformly 30 too high guides perfectly and scores terribly on RMSE.
Checkpoints are selected on top-1 accuracy for the same reason. A small
regression term stays only to anchor a scale, which pure ranking leaves
undefined and weighted A* needs.
**Nodes expanded is not differentiable**, so the reinforcement stage
reaches it three ways: DAgger for the distribution shift, bootstrapping
for instances too hard to label, and the cross-entropy method over the
weight vector with the planner as a black box.
Two findings worth recording, both measured rather than reasoned:
- CEM must tune on instances with headroom. On the training ladder the
imitated heuristic already expands about as many nodes as the plan is
long, so every perturbation scores the same. Tuning there moved the
score 12.83 -> 12.75; tuning a rung higher moved it 1605 -> 64.
- CEM must select on instances it is not fitting. The first version
selected on its tuning set, reported 108 expansions, and scored 1734
on held-out instances -- nearly five times worse than the imitated
heuristic it started from. A thousand parameters had been fitted to
eight instances and had fitted them. Selecting the incumbent on a
disjoint instance family brings the same command to 137.
It does not always win. On logistics it loses to hff by 6x, and the
reason is exact: that domain has two predicates, so the feature vector
is eight numbers and cannot distinguish which package is where, only how
many are somewhere. Two states with a package at its destination and
across the map are identical under it. That is the argument for
relational representations stated as a measurement.
The learning stack is stdlib-only like the rest of the core -- verified
by training in an environment with no numpy installed. The numpy path is
a speed option worth one to two orders of magnitude, and a test pins the
two implementations to identical gradients, both checked against finite
differences.
`learned:<model.json>` resolves anywhere a heuristic name is accepted,
via a lazy loader, so nothing in the core imports the learning stack and
a planner that never asks for one never pays for it.
.docs/ carries the research notes: prior work, the measured results
including the failure, the MDP the RL stage corresponds to, and what to
build next.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Meb35zHKsyBkH2sbWoyKMT
Correction to a claim in the PR descriptionWhile building a promo video for this work — which re-measures everything rather than quoting the notes — one of the headline claims failed to reproduce. It was wrong, and both the description and What the description said
What is actually trueTwo things changed in the same edit: the validation split went in, and the perturbation scale
The validation split makes no measurable difference here. Which is the exact mistake I'd written a code comment warning about, one commit earlier, in the collection script for this video. And the mean was doing the lyingPer instance, held-out set:
Nine of ten instances improve under both settings by roughly the same factor. The entire 1734-vs-137 gap is one instance. What survivesThe substantive results are unaffected:
No code changes; the mechanism is sound. Generated by Claude Code |
`tools/make_learn_promo.py` renders a 97-second tour of the learned
heuristic and the reinforcement stage. Like the main promo it measures
everything at render time -- it trains, runs CEM, and re-runs both
failure modes -- so the video cannot drift from the notes. That is not
decoration: building it is what caught the errors below.
Eleven scenes: a plan handing over its own labels, imitation, why the
ordering is the thing search reads, the MDP, CEM descending, and then
the two traps, the result, and the domain where it loses.
Two corrections to claims already published in .docs/ and on the PR.
**The validation split was not what improved transfer.** Two settings
changed in one edit -- the split went in, and sigma went 0.05 -> 0.15 --
and the improvement was credited entirely to the first. Varying one at a
time:
sigma 0.05, selected on the tuning set 1734
sigma 0.05, selected on a disjoint family 1730
sigma 0.15, selected on the tuning set 137
sigma 0.15, selected on a disjoint family 137
sigma was doing all of it. Widening the validation family to span sizes
past the tuning range does not change that either. The split is still
correct and still worth its one extra scoring pass -- it bounds what can
be returned, and took validation 276 -> 73 on this run -- but it is a
guardrail, not the knob, and I should not have credited it with someone
else's result.
**And the mean was doing the lying.** Per instance on the held-out set,
nine of ten improve under both settings by roughly the same factor. The
entire 1734-versus-137 gap is blocksworld-13-7777, which imitation could
not solve inside 30000 expansions at all. A mean over that distribution
is close to a report of one instance.
What survives is the coverage claim, which was always the strongest one:
imitation fails that instance, both tuned versions solve it, and the
median improvement across the set is about 1.4x.
The general lesson is the ordinary one, which the derivative-free
framing made easy to forget: change one thing at a time, and look at the
distribution before believing the mean. It is now in AGENTS.md so the
next person does not have to rediscover it.
promo/rl-data.json caches the measurement pass; delete it to re-measure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Meb35zHKsyBkH2sbWoyKMT
CodeFactor flagged one issue on the previous commit: `collect()` at 61 statements against a limit of 50, with 54 locals. It was the only pylint rule class present in make_learn_promo.py and not already in make_promo.py, which is how it was identified -- CodeFactor's own report needs an account to read. The complaint was fair. One function was building four instance families, annotating a plan, training, reinforcing, reproducing two failure modes and measuring a second domain. It is now six functions that each do one of those, plus an orchestrator that reads as a summary of what the video contains. `_transfer` also collapses six near-identical `evaluate_transfer` calls that were repeating the same budget and time limit; those are now module-level constants, so changing the budget changes it everywhere rather than in five of six places. Behaviour is unchanged, and checked rather than assumed: re-running the full measurement pass produces a byte-identical cache apart from wall-clock timings, which vary run to run. Every expansion count, coverage figure and plan cost matches. The video is not re-rendered -- the scene functions were not touched and the data behind them is the same. No remaining pylint rule class is unique to this file relative to the promo renderer already on main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Meb35zHKsyBkH2sbWoyKMT
Three markdownlint findings in the files this branch adds: two fenced blocks in .docs/rl-for-search.md with no language, and one block in .docs/README.md indented where the rest of the file fences. Fixed on their own merits -- a fence without a language gets no highlighting and no copy affordance on GitHub. Not claimed as the CodeFactor fix. Its report needs an account to read and I could not reproduce a single-issue result locally with pylint, bandit or markdownlint at their defaults, all of which return far more. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Meb35zHKsyBkH2sbWoyKMT
On the CodeFactor statusFlagging this rather than silently chasing it: CodeFactor's report is not readable without an account (both I tried to identify it by reproducing locally and could not, because every analyzer CodeFactor plausibly runs returns far more than one issue on this diff at its defaults:
So whatever ruleset CodeFactor uses is a curated subset I can't infer from a single integer. Two things I did fix
Neither is claimed as the fix. Both are improvements on their own merits. Why I am not driving this one to greenBecause red CodeFactor appears to be this repository's steady state rather than a regression signal:
I do not think it is right for me to keep shotgunning speculative changes at an opaque count, and I would rather not touch If someone with CodeFactor access pastes the finding here, I will fix it directly. Everything else on the PR is green: Generated by Claude Code |
Found reviewing this branch before merge. `_default_labeller` caches one hff instance per task, keyed on `id(task)` because `Task` is an unfrozen dataclass and therefore unhashable. Nothing held a reference to the task, and CPython reuses addresses once an object is collected -- so a caller passing tasks the cache does not keep alive could be handed another task's relaxed-task tables. That would not raise. It would quietly mislabel the aggregated samples, which is the failure mode this whole module is least able to notice. Unreachable as things stand: `dagger` holds its task list for the whole loop. But `labeller` is a public parameter, and the property that makes this safe lives in a different function from the code that depends on it. The entry now holds the task, which makes the id un-recyclable while the entry lives, plus an identity check so it stays correct even if that reasoning is ever wrong. Test replays each returned plan against the task it was requested for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Meb35zHKsyBkH2sbWoyKMT
ReviewReviewed by risk rather than line by line: the newest logic ( One issue found and fixed — 684f5b0
That would not raise. It would silently mislabel the aggregated DAgger samples, which is precisely the failure this module is least equipped to notice: the labels look plausible, training succeeds, and the heuristic is quietly worse. Unreachable as things stand — What holds up
Smaller notes, not blocking
On CodeFactorRed, at "1 issue found." I could not identify it — the report is 403 without an account, and pylint/bandit/markdownlint all return far more than one on this diff at their defaults, so the ruleset is a curated subset I cannot infer from an integer. Details and what I fixed anyway are in the comment above. Worth stating plainly for the record: #151 was merged at "11 issues found." Red CodeFactor is this repository's steady state, not a signal this branch regressed something. If someone with access pastes the finding, I will fix it. VerdictApprove and merge. 415 tests green on Python 3.9 through 3.14, The results are stated with their limits attached: the headline 137 is a mean over a heavy tail dominated by one instance, the strongest claim is the coverage one, and the domain where this loses has its mechanism spelled out rather than glossed. Two earlier claims that did not survive re-measurement are corrected in place rather than deleted. Generated by Claude Code |
2 participants
Every solved instance is already a labelled trajectory: the cost of a plan's suffix from a state on it is that state's cost-to-go.
jupyddl learnturns a corpus of those into a heuristic, then stops imitatingh*and starts optimising the thing that actually matters — the number of nodes search expands.jupyddl learn blocksworld --sizes 3-6 --seeds-per-size 3 \ --cem 10 --cem-sizes 9-12 --evaluate 9-13 -o bw.heur.json jupyddl solve domain.pddl problem.pddl -s gbfs -H learned:bw.heur.jsonResult
Trained on 3–6 block instances, evaluated on 9–13 block instances from a seed family no stage of training saw, under greedy best-first search:
learnedhffgoalcountblind3.8× fewer expansions than
hffand 15× faster. About a minute end to end on one CPU.Read that mean with care. The held-out set has a heavy tail: nine of ten instances sit between 58 and 227 expansions, and the tenth (
blocksworld-13-7777) moves the average on its own. The median improvement over imitation is ~1.4×. The most defensible single claim is the coverage one — imitation could not solve that instance inside 30 000 expansions and the tuned heuristic solves it in 214.Three design decisions did most of the work
Features are keyed on the predicate symbol, never the ground atom, and normalised per symbol. That is the entire transfer story — a one-hot over ground atoms changes length and the meaning of every slot with each instance, so a model trained on four blocks cannot even be evaluated on forty.
The objective is ranking, not regression. GBFS never reads a heuristic value, only the order it imposes: a model uniformly 30 too high guides perfectly while scoring terribly on RMSE, and a model with excellent RMSE that inverts two siblings sends the search into the wrong subtree. Checkpoints are selected on top-1 accuracy for the same reason. (Chrestien et al., NeurIPS 2023.)
Nodes expanded is not differentiable — it comes out the far side of a priority queue — so the reinforcement stage reaches it three ways: DAgger for the distribution shift, bootstrapping for instances too hard to label, and the cross-entropy method over the weight vector with the planner as a black box.
What the measurements actually showed
CEM must tune on instances with headroom. On the training ladder the imitated heuristic already expands about as many nodes as the plan is long, so every perturbation scores identically and the objective is flat. Tuning there moved the score 12.8 → 12.2 — noise. Tuning a rung higher moved 152 → 101.
The perturbation scale is the knob that matters. σ=0.05 and σ=0.15 differ by an order of magnitude in held-out cost (1734 vs 137) on otherwise identical runs, and the difference is concentrated entirely in the single hardest instance.
Selecting the incumbent on a disjoint instance family is a guardrail, not the knob. It guarantees the returned model is no worse on instances the optimiser did not fit (validation 276 → 73 on this run), and it costs one scoring pass per iteration. It is worth keeping. It is not what produced the headline number, and an earlier version of this description said it was.
Both scores are printed every iteration, so a run fitting its tuning set while losing validation is visible rather than silent.
It does not always win
On logistics it loses to
hffby 6× (204 expansions vs 35), and the reason is exact rather than mysterious: that domain has two predicates, so the feature vector is eight numbers and cannot distinguish which package is where, only how many are somewhere. Two states with a package at its destination and across the map are identical under it. Top-1 accuracy 0.656.That is the argument for relational representations stated as a measurement, and it is the first item on the roadmap.
Integration
learned:<model.json>resolves anywhere a heuristic name is accepted —solve,benchmark, the API — through a lazy loader, so nothing in the core imports the learning stack and a planner that never asks for one never pays for it.make_heuristicalso passes an already-built heuristic through, so callers holding a trained model need not round-trip it to disk.The learning stack is stdlib-only like the rest of the core, verified by training in an environment with no NumPy installed. The
learnextra adds NumPy purely for speed (one to two orders of magnitude); a test pins the two implementations to identical gradients, and both are checked against finite differences.Verification
LearnedHeuristic.admissibleisFalseand the docs say so: nothing in the objective bounds the prediction from above, so pair it withgbfsorwastar, never with an optimality claim.Research notes
.docs/carries the write-up: prior work and where this sits in it, the measured results including the logistics failure, the MDP the RL stage corresponds to and why the obvious policy gradient is harder than it looks, the corrections above with the per-instance data behind them, and a roadmap ordered by expected value.